Add Anthropic Message Batches support - #6713
Open
ricken07 wants to merge 7 commits into
Open
Conversation
Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Add AnthropicBatchModel, exposing the five provider operations of the Message Batches API, and its default implementation over the official Anthropic Java SDK. Batch processing is asynchronous and can take up to 24 hours, so this is deliberately not a ChatModel: submit has no immediate ChatResponse to return. The control operations are synchronous; only results is reactive, backed by Flux.using so the SDK stream is closed on termination and on cancellation and a batch with a very large number of entries is never held in memory. Entries are mapped through the same Prompt conversion as AnthropicChatModel.call, so prompt caching, thinking, images and PDF documents, structured output and tool definitions behave identically on both paths, and a succeeded result is converted back into a ChatResponse with the same generations, metadata and usage. Tool calls are not executed: a batch entry cannot be continued mid-flight, so a tool_use response is returned as is. Spring AI provides the provider access, not the orchestration. There is no polling, no persistence and no scheduling: those stay with the application, which owns the batch id and the customId correlations. Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Expose the batch model as a Spring Boot bean so an application can inject it without building an SDK client itself. The bean is opt-in behind spring.ai.anthropic.batch.enabled, so applications that never submit batches do not pay for a second HTTP client. Connection settings and the chat model defaults are reused as-is, which means a batch shares the credentials, base URL, timeout, retries, proxy, custom headers and HTTP client customizers of realtime calls; spring.ai.anthropic.batch.model and .max-tokens override the model and output ceiling for batch entries only. Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Cover the submit, poll, read, cancel and error paths against the real API, which the mocked unit tests cannot validate: the JSONL result stream, the out-of-order correlation by customId and the usage reported per entry. Two switches guard it, because a batch consumes quota and its completion time is not bounded by the API contract: ANTHROPIC_API_KEY must be set, the repository-wide convention, and ANTHROPIC_BATCH_IT_DISABLED turns it off even when a key is present. A batch is asynchronous by design, so a silent wait would be indistinguishable from a hung test. Each poll logs the elapsed time, the status and the counters, and past the budget the test aborts rather than fails: a slow queue on Anthropic's side is not a regression here. Only one test waits, so the suite waits once. Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
Add a dedicated anthropic-batch.adoc page rather than growing anthropic-chat.adoc, which was already the largest page under api/chat at over 1700 lines. It is nested under the Anthropic entry in the navigation, the shape the embeddings section already uses for Bedrock and VertexAI, and anthropic-chat.adoc keeps a short pointer so the feature stays discoverable from the provider's entry page. The page covers submitting a batch, polling it, streaming the results, cancelling and deleting, the auto-configuration properties, and the two constraints callers have to design around: correlation by customId because results are unordered, and the absence of an interactive tool-execution loop. Signed-off-by: Ricken BAZOLO <ricken07@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
spring-ai-anthropicmodule exposesAnthropicChatModelfor synchronous and streaming calls, but there is no abstraction for Anthropic's Message Batches API. Applications that need bulk processing at reduced cost have to create a second Anthropic SDK client themselves and pin their own SDK version, which drifts from Spring AI on HTTP configuration, headers and observability.This adds
AnthropicBatchModelalongside the chat model, covering the five provider operations: submit, retrieve, results, cancel and delete.